{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9aaf39be-eca2-4f26-b469-0499a1b52648",
   "metadata": {},
   "source": [
    "# Step-wise, Controllable Agents\n",
    "\n",
    "This notebook shows you how to use our brand-new lower-level agent API, which supports a host of functionalities beyond simply executing a user query to help you create tasks, iterate through steps, and control the inputs for each step.\n",
    "\n",
    "### High-Level Agent Architecture\n",
    "\n",
    "Our \"agents\" are composed of `AgentRunner` objects that interact with `AgentWorkers`. `AgentRunner`s are orchestrators that store state (including conversational memory), create and maintain tasks, run steps through each task, and offer the user-facing, high-level interface for users to interact with.\n",
    "\n",
    "`AgentWorker`s **control the step-wise execution of a Task**. Given an input step, an agent worker is responsible for generating the next step. They can be initialized with parameters and act upon state passed down from the Task/TaskStep objects, but do not inherently store state themselves. The outer `AgentRunner` is responsible for calling an `AgentWorker` and collecting/aggregating the results.\n",
    "\n",
    "If you are building your own agent, you will likely want to create your own `AgentWorker`. See below for an example!\n",
    "\n",
    "### Notebook Walkthrough\n",
    "\n",
    "This notebook shows you how to run step-wise execution and full-execution with agents. \n",
    "- We show you how to do execution with OpenAIAgent (function calling)\n",
    "- We show you how to do execution with ReActAgent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87bd57f6-3804-48b4-b53a-9815a6dfc48c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f202c5bb-3cd1-46ab-9b66-1bc28c6012b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from typing import Sequence, List\n",
    "\n",
    "from llama_index.llms import OpenAI, ChatMessage\n",
    "from llama_index.tools import BaseTool, FunctionTool\n",
    "\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "289364ec-a9d2-47e5-86c2-44a517fb4f03",
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiply(a: int, b: int) -> int:\n",
    "    \"\"\"Multiple two integers and returns the result integer\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "multiply_tool = FunctionTool.from_defaults(fn=multiply)\n",
    "\n",
    "\n",
    "def add(a: int, b: int) -> int:\n",
    "    \"\"\"Add two integers and returns the result integer\"\"\"\n",
    "    return a + b\n",
    "\n",
    "\n",
    "add_tool = FunctionTool.from_defaults(fn=add)\n",
    "\n",
    "tools = [multiply_tool, add_tool]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffe1a3fb-9224-485d-be0b-7dbb5358c4ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = OpenAI(model=\"gpt-3.5-turbo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62afda27-5191-4719-b8c7-0189d745ef05",
   "metadata": {},
   "source": [
    "## Test OpenAI Agent\n",
    "\n",
    "There's two main ways to initialize the agent.\n",
    "- **Option 1**: Initialize `OpenAIAgent`. This is a simple subclass of `AgentRunner` that bundles the `OpenAIAgentWorker` under the hood.\n",
    "- **Option 2**: Initialize `AgentRunner` with `OpenAIAgentWorker`. Here you import the modules and compose your own agent.\n",
    "\n",
    "**NOTE**: The old OpenAIAgent can still be imported via `from llama_index.agent import OldOpenAIAgent`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1452368b-9f3a-4da6-8c5e-11266250f6b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.agent import AgentRunner, OpenAIAgentWorker, OpenAIAgent\n",
    "\n",
    "# Option 1: Initialize OpenAIAgent\n",
    "agent = OpenAIAgent.from_tools(tools, llm=llm, verbose=True)\n",
    "\n",
    "\n",
    "# # Option 2: Initialize AgentRunner with OpenAIAgentWorker\n",
    "# openai_step_engine = OpenAIAgentWorker.from_tools(tools, llm=llm, verbose=True)\n",
    "# agent = AgentRunner(openai_step_engine)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fe41cdb-1107-443e-8c79-f06d95fb9621",
   "metadata": {},
   "source": [
    "### Test E2E Chat\n",
    "\n",
    "Here we re-demonstrate the end-to-end execution of a user task through the `chat()` function.\n",
    "\n",
    "This will iterate step-wise until the agent is done with the current task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f422d9a7-ee49-4af7-bd73-6c5e3c8747ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AgentChatResponse(response='Hello! How can I assist you today?', sources=[], source_nodes=[])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agent.chat(\"Hi\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d957d19d-a6a4-42f0-adfa-2a8370e1c296",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Calling Function ===\n",
      "Calling function: multiply with args: {\n",
      "  \"a\": 121,\n",
      "  \"b\": 3\n",
      "}\n",
      "Got output: 363\n",
      "========================\n",
      "\n",
      "=== Calling Function ===\n",
      "Calling function: add with args: {\n",
      "  \"a\": 363,\n",
      "  \"b\": 42\n",
      "}\n",
      "Got output: 405\n",
      "========================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "response = agent.chat(\"What is (121 * 3) + 42?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1776f7fa-b24a-47e7-a265-f71bbf80479f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AgentChatResponse(response='The result of (121 * 3) + 42 is 405.', sources=[ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363), ToolOutput(content='405', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 363, 'b': 42}}, raw_output=405)], source_nodes=[])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82983fbd-15ce-4620-aceb-d61cd8765597",
   "metadata": {},
   "source": [
    "### Test Step-Wise Execution\n",
    "\n",
    "Now let's show the lower-level API in action. We do the same thing, but break this down into steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1443a4f5-3335-443d-81e3-bb6fd2e0ddd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# start task\n",
    "task = agent.create_task(\"What is (121 * 3) + 42?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53c47b0b-6b60-44a8-85fc-5354779ab372",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Calling Function ===\n",
      "Calling function: multiply with args: {\n",
      "  \"a\": 121,\n",
      "  \"b\": 3\n",
      "}\n",
      "Got output: 363\n",
      "========================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1faf5271-37ba-4c23-b9d7-de593581ef16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TaskStepOutput(output=AgentChatResponse(response='None', sources=[ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363)], source_nodes=[]), task_step=TaskStep(task_id='920e1e67-2310-430b-bfe5-813f48c9c23c', step_id='697f90f1-f6f2-4be5-a9f9-a7239eadc094', input='What is (121 * 3) + 42?', step_state={}, next_steps={}, prev_steps={}, is_ready=True), next_steps=[TaskStep(task_id='920e1e67-2310-430b-bfe5-813f48c9c23c', step_id='e526d116-0f44-4337-915f-be55a04511b9', input=None, step_state={}, next_steps={}, prev_steps={}, is_ready=True)], is_last=False)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "step_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61083776-1de3-453a-98b2-5d0d235af8f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Calling Function ===\n",
      "Calling function: add with args: {\n",
      "  \"a\": 363,\n",
      "  \"b\": 42\n",
      "}\n",
      "Got output: 405\n",
      "========================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cc8e987-b895-4689-b8c5-dc3646e0d719",
   "metadata": {},
   "outputs": [],
   "source": [
    "step_output = agent.run_step(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc1152d8-7e5e-4059-a9ab-b3ddaa86a6e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "# display final response\n",
    "print(step_output.is_last)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca25c63f-19e3-4a59-b91d-47ac6961ace7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The result of (121 * 3) + 42 is 405.\n"
     ]
    }
   ],
   "source": [
    "# now that the step execution is done, we can finalize response\n",
    "response = agent.finalize_response(task.task_id)\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0faaf68e-55e1-4af5-9dd0-fb4248768dc3",
   "metadata": {},
   "source": [
    "## Test ReAct Agent\n",
    "\n",
    "We do the same experiments, but with ReAct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de8d1111-c603-4bff-94b1-631cc4bdf134",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = OpenAI(model=\"gpt-4-1106-preview\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3aef2609-3227-435f-a370-aae0e9413095",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.agent import AgentRunner, ReActAgentWorker, ReActAgent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecbab00e-0227-4770-a543-0c022030e303",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 1: Initialize OpenAIAgent\n",
    "agent = ReActAgent.from_tools(tools, llm=llm, verbose=True)\n",
    "\n",
    "# # Option 2: Initialize AgentRunner with ReActAgentWorker\n",
    "# react_step_engine = ReActAgentWorker.from_tools(tools, llm=llm, verbose=True)\n",
    "# agent = AgentRunner(react_step_engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33d79713-86f6-4dbb-948f-6c4a84072e95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;5;200mThought: The user has greeted me, and I should respond in kind.\n",
      "Response: Hello! How can I assist you today?\n",
      "\u001b[0m"
     ]
    },
    {
     "data": {
      "text/plain": [
       "AgentChatResponse(response='Hello! How can I assist you today?', sources=[], source_nodes=[])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agent.chat(\"Hi\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec787399-59c5-4fa1-8068-b9f5847f5ba8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;5;200mThought: I need to use a tool to help me calculate the multiplication part of the question first.\n",
      "Action: multiply\n",
      "Action Input: {'a': 121, 'b': 3}\n",
      "\u001b[0m\u001b[1;3;34mObservation: 363\n",
      "\u001b[0m\u001b[1;3;38;5;200mThought: Now that I have the result of the multiplication, I need to add 42 to it.\n",
      "Action: add\n",
      "Action Input: {'a': 363, 'b': 42}\n",
      "\u001b[0m\u001b[1;3;34mObservation: 405\n",
      "\u001b[0m\u001b[1;3;38;5;200mThought: I can answer without using any more tools.\n",
      "Response: (121 * 3) + 42 equals 405.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "response = agent.chat(\"What is (121 * 3) + 42?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a13e93f5-415d-48cc-8966-3212b2e89c51",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AgentChatResponse(response='(121 * 3) + 42 equals 405.', sources=[ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363), ToolOutput(content='405', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 363, 'b': 42}}, raw_output=405)], source_nodes=[])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "696fd120-a24f-4bb8-aaa6-c0a3a518762b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# start task\n",
    "task = agent.create_task(\"What is (121 * 3) + 42?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db2bf7cf-1092-44a2-bb9a-f23230b61323",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;5;200mThought: I need to use a tool to help me answer the question.\n",
      "Action: multiply\n",
      "Action Input: {'a': 121, 'b': 3}\n",
      "\u001b[0m\u001b[1;3;34mObservation: 363\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eeb952a4-bebd-4f31-b299-bf9899b5fed4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AgentChatResponse(response='Observation: 363', sources=[ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363)], source_nodes=[])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "step_output.output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ab3c9e2-073d-4390-ad07-b5a1bcdef2c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;5;200mThought: Now that I have the result of the multiplication, I need to add 42 to it.\n",
      "Action: add\n",
      "Action Input: {'a': 363, 'b': 42}\n",
      "\u001b[0m\u001b[1;3;34mObservation: 405\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62f12f48-eb91-4868-a5ad-20fa24bce3f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AgentChatResponse(response='Observation: 405', sources=[ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363), ToolOutput(content='405', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 363, 'b': 42}}, raw_output=405)], source_nodes=[])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "step_output.output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26f44979-1c89-4433-994b-a295a236dd76",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;5;200mThought: I can answer without using any more tools.\n",
      "Response: (121 * 3) + 42 equals 405.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33ee252f-f743-4ae7-bf44-c755d6c280e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AgentChatResponse(response='(121 * 3) + 42 equals 405.', sources=[ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363), ToolOutput(content='405', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 363, 'b': 42}}, raw_output=405)], source_nodes=[])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "step_output.output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f57cf1a9-0a64-4b2c-942f-c378455de007",
   "metadata": {},
   "source": [
    "### List Out Tasks\n",
    "\n",
    "There are 3 tasks, corresponding to the three runs above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebc5349f-b929-4cfa-97e8-651b24844046",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "tasks = agent.list_tasks()\n",
    "print(len(tasks))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecc9ba9e-ff1e-439c-9d7c-dab72a862191",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'What is (121 * 3) + 42?'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_state = tasks[-1]\n",
    "task_state.task.input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c83513bf-f285-4af2-9c62-04283294f9e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get completed steps\n",
    "completed_steps = agent.get_completed_steps(task_state.task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "776b2c70-2c2b-49d3-aaf8-b15049eb3135",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(completed_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a37f359b-28bb-4eef-9293-bb6f8a6c3877",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TaskStepOutput(output=AgentChatResponse(response='Observation: 363', sources=[ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363), ToolOutput(content='405', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 363, 'b': 42}}, raw_output=405)], source_nodes=[]), task_step=TaskStep(task_id='1a71f7cb-c854-4baa-ad11-0c97460a6af2', step_id='72ed54f4-4edd-496b-bb6a-d58f4275f03e', input='What is (121 * 3) + 42?', step_state={}, next_steps={}, prev_steps={}, is_ready=True), next_steps=[TaskStep(task_id='1a71f7cb-c854-4baa-ad11-0c97460a6af2', step_id='6486cc4a-15ea-490a-b761-9dc88881fc24', input=None, step_state={}, next_steps={}, prev_steps={}, is_ready=True)], is_last=False)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "completed_steps[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88f10cb1-3133-4c77-a338-1fa6ce86b0f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 0\n",
      "Response: Observation: 363\n",
      "Sources: [ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363), ToolOutput(content='405', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 363, 'b': 42}}, raw_output=405)]\n",
      "Step 1\n",
      "Response: Observation: 405\n",
      "Sources: [ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363), ToolOutput(content='405', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 363, 'b': 42}}, raw_output=405)]\n",
      "Step 2\n",
      "Response: (121 * 3) + 42 equals 405.\n",
      "Sources: [ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363), ToolOutput(content='405', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 363, 'b': 42}}, raw_output=405)]\n"
     ]
    }
   ],
   "source": [
    "for idx in range(len(completed_steps)):\n",
    "    print(f\"Step {idx}\")\n",
    "    print(f\"Response: {completed_steps[idx].output.response}\")\n",
    "    print(f\"Sources: {completed_steps[idx].output.sources}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama_index_logan",
   "language": "python",
   "name": "llama_index_logan"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
